Assessing the research landscape and clinical utility of large language models: a scoping review.

IMPORTANCE: Large language models (LLMs) like OpenAI's ChatGPT are powerful generative systems that rapidly synthesize natural language responses. Research on LLMs has revealed their potential and pitfalls, especially in clinical settings. However, the evolving landscape of LLM research in medicine has left several gaps regarding their evaluation, application, and evidence base. OBJECTIVE: This scoping review aims to (1) summarize current research evidence on the accuracy and efficacy of LLMs in medical applications, (2) discuss the ethical, legal, logistical, and socioeconomic implications of LLM use in clinical settings, (3) explore barriers and facilitators to LLM implementation in healthcare, (4) propose a standardized evaluation framework for assessing LLMs' clinical utility, and (5) identify evidence gaps and propose future research directions for LLMs in clinical applications. EVIDENCE REVIEW: We screened 4,036 records from MEDLINE, EMBASE, CINAHL, medRxiv, bioRxiv, and arXiv from January 2023 (inception of the search) to June 26, 2023 for English-language papers and analyzed findings from 55 worldwide studies. Quality of evidence was reported based on the Oxford Centre for Evidence-based Medicine recommendations. FINDINGS: Our results demonstrate that LLMs show promise in compiling patient notes, assisting patients in navigating the healthcare system, and to some extent, supporting clinical decision-making when combined with human oversight. However, their utilization is limited by biases in training data that may harm patients, the generation of inaccurate but convincing information, and ethical, legal, socioeconomic, and privacy concerns. We also identified a lack of standardized methods for evaluating LLMs' effectiveness and feasibility. CONCLUSIONS AND RELEVANCE: This review thus highlights potential future directions and questions to address these limitations and to further explore LLMs' potential in enhancing healthcare delivery.

Mitigating diagnostic performance bias in a skin-tone balanced dermatology curriculum.

INTRODUCTION: Individuals with skin of colour (SoC) have delayed diagnosis and poorer outcomes when presenting with some dermatologic conditions when compared to individuals with light skin (LS). The objective of this study was to determine if diagnostic performance bias can be mitigated by a skin-tone balanced dermatology curriculum. METHODOLOGY: A prospective randomised intervention study occurred over 2 weeks in 2020 at a Canadian medical school. A convenience sample of all first-year medical students (n = 167) was chosen. In week 1, all participants had access to dermatology podcasts and were randomly allocated to receive non-analytic training (NAT; online patient 'cards') on either SoC cases or LS cases. In week 2, all participants received combined training (CT; NAT and analytic training through workshops on how to apply dermatology diagnostic rules for all skin tones). Participating students completed two formative assessments after weeks 1 and 2. RESULTS: Ninety-two students participated in the study. After week 1, both groups had a lower diagnostic performance on SoC (p = 0.0002 and p = 0.002 for students who trained on LS 'cards' and SoC 'cards', respectively). There was a significant decrease in mean skin tone difference in both groups after week 2 (initial training on SoC: 5.8% (SD 12.2) pre, -1.4% (14.7) post, p = 0.007; initial training on LS: 7.8% (15.4) pre, -4.0% (11.8%) post, p = 0.0001). Five students participated in a post-study survey in 2023, and all found the curriculum enhanced their diagnostic skills in SoC. CONCLUSIONS: SoC performance biases of medical students disappeared after CT in a skin tone-balanced dermatology curriculum.

Wellbeing and mental health amongst medical students in Canada.

RESEARCH: There is abundant data revealing that there is significant rate of rates of Psychiatric morbidity, psychological stress, and burnout in the medical student population. A core study group in the UK collaborated with 12 countries around the world to review medical student wellness. In this context we surveyed 101 medical students at the Cummings medical school, Calgary, Canada during the height of the COVID pandemic regarding their wellbeing and mental health. RESULTS/MAIN FINDINGS: Prior to medical school 27% reported a diagnosis with a mental disorder. Whilst at medical school 21% reported a mental health condition, most commonly an anxiety disorder and or depressive disorder. The most commonly reported source of stress was study at 81%, the second being relationships at 62%, money stress was a significant source of stress for 35%, and finally 10% reported accommodation or housing as stressful. Interestingly only 14% tested CAGE positive but 20% of students reported having taken a non-prescription substance to feel better or regulate their mood. Seventy-five percent of medical students met specific case criteria for exhaustion on the Oldenburg Burnout inventory 74% met criteria for the GHQ questionnaire. CONCLUSIONS: These findings confirm that medical students are facing significant stressors during their training. These stressors include, in order of frequency, study, relational, financial, and accommodation issues. Nonprescription Substance use was a common finding as well as exhaustion and psychiatric morbidity. Future interventions pursued will have to address cultural issues as well as the organizational and individual determinates of stress.

Exploring the global impact of the COVID-19 pandemic on medical education: an international cross-sectional study of medical learners.

BACKGROUND: The evidence surrounding the impact of COVID-19 on medical learners remains anecdotal and highly speculative despite the anticipated impact and potential consequences of the current pandemic on medical training. The purpose of this study was to explore the extent that COVID-19 initially impacted medical learners around the world and examine global trends and patterns across geographic regions and levels of training. METHODS: A cross-sectional survey of medical learners was conducted between March 25-June 14, 2020, shortly after the World Health Organization declared COVID-19 a pandemic. RESULTS: 6492 learners completed the survey from 140 countries. Most medical schools removed learners from the clinical environment and adopted online learning, but students reported concerns about the quality of their learning, training progression, and milestone fulfillment. Residents reported they could be better utilized and expressed concerns about their career timeline. Trainees generally felt under-utilized and wanted to be engaged clinically in meaningful ways; however, some felt that contributing to healthcare during a pandemic was beyond the scope of a learner. Significant differences were detected between levels of training and geographic regions for satisfaction with organizational responses as well as the impact of COVID-19 learner wellness and state-trait anxiety. CONCLUSIONS: The disruption to the status quo of medical education is perceived by learners across all levels and geographic regions to have negatively affected their training and well-being, particularly amongst postgraduate trainees. These results provide initial empirical insights into the areas that warrant future research as well as consideration for current and future policy planning.

CONTEXTE: On s'attendait à ce que la pandémie de la COVID-19 ait des conséquences sur la formation médicale, mais les constats relatifs à son impact sur les étudiants en médecine demeurent anecdotiques et plutôt spéculatifs. L'objectif de cette étude était d'explorer l'étendue des premiers effets de la COVID-19 sur les étudiants en médecine dans le monde et d'examiner les tendances et les schémas qui se dégagent, quels que soient la région géographique ou le niveau d'études. MÉTHODES: Une enquête transversale sur les étudiants en médecine a été menée entre le 25 mars et le 14 juin 2020, peu après que l'Organisation mondiale de la santé ait déclaré que la COVID-19 était une pandémie. RÉSULTATS: Le sondage a été réalisé auprès de 6492 étudiants de 140 pays. La plupart des facultés de médecine ont retiré les apprenants de l'environnement clinique et adopté l'apprentissage en ligne, mais les étudiants ont exprimé des préoccupations quant à sa qualité, à la progression de la formation et à l'atteinte de divers jalons. Les résidents jugent qu'ils pourraient être plus utiles et s'inquiètent de l'avancement de leur carrière. Les apprenants se sentent généralement sous-utilisés et souhaitent s'engager cliniquement de manière plus significative; cependant, certains estiment qu'il n'est pas à propos de demander aux étudiants de contribuer aux soins de santé pendant une pandémie. Des écarts importants ont été relevés entre les différents niveaux de formation et les différentes régions géographiques en ce qui concerne la satisfaction face aux réponses organisationnelles, l'impact de la COVID-19 sur leur bien-être et l'anxiété chronique et réactionnelle. CONCLUSIONS: La perturbation du statu quo dans l'éducation médicale est perçue par les étudiants de tous les niveaux et de toutes les régions géographiques, mais davantage encore par les résidents, comme ayant affecté négativement et leur formation et leur bien-être. Ces résultats fournissent des aperçus empiriques préliminaires sur les domaines qui méritent des recherches futures et qui devraient être pris en compte dans la formulation des politiques actuelles et à venir.

Exploring the impact of the COVID-19 pandemic on medical learner wellness: a needs assessment for the development of learner wellness interventions.

BACKGROUND: On March 11, 2020 the World Health Organization declared the novel coronavirus SARS-CoV-2 disease (COVID-19) a global pandemic. We sought to understand impact of COVID-19 on learner wellness at a large tertiary care academic institution to inform the future development of learner wellness interventions during the COVID-19 pandemic. METHODS: A cross-sectional, internet-based survey collected quantitative and qualitative data from learners April-June 2020. Descriptive statistics and univariate analyses were reported for quantitative data. Open-ended, qualitative responses were analyzed deductively using thematic analysis. RESULTS: Twenty percent of enrolled learners in that faculty of medicine (540/2741) participated including undergraduate [Bachelor's] students (25.7%), graduate [science] students (27.5%), undergraduate medical students (22.8%), and postgraduate resident physicians (23.5%). We found that learner wellness across all stages of training was negatively impacted and the ways in which learners were impacted varied as a result of their program's response to the COVID-19 pandemic. CONCLUSIONS: Learners in health sciences and medical education report worsening well-being because of the programs and the systems in which they function with the added burden of the COVID-19 pandemic. Future interventions would benefit from a holistic framework of learner wellness while engaging in systems thinking to understand how individuals, programs and respective systems intersect. The importance of acknowledging equity, diversity and inclusion, fostering psychological safety and engaging learners as active participants in their journey during a pandemic and beyond are key elements in developing wellness interventions.

CONTEXTE: Le 11 mars 2020, l'Organisation mondiale de la santé a déclaré que le nouveau coronavirus SRAS-CoV-2 (COVID-19) était pandémique. Nous avons tenté de cerner l'impact de la COVID-19 sur le bien-être des apprenants dans un grand centre universitaire de soins tertiaires afin d'étayer le développement futur d'interventions en faveur du bien-être des apprenants en contexte de pandémie. MÉTHODES: Une enquête transversale par sondage en ligne, menée entre les mois d'avril et juin 2020, a permis de recueillir des données quantitatives et qualitatives auprès des apprenants. Des statistiques descriptives et des analyses univariées ont été présentées comme données quantitatives. Les réponses ouvertes, qualitatives, ont été analysées de manière déductive par le biais d'une analyse thématique. RÉSULTATS: Vingt pour cent des étudiants de la faculté de médecine (540/2741) ont participé, dont 25,7 % étaient inscrits au premier cycle (baccalauréat), 27,5 % au deuxième cycle (sciences), 22,8 % au premier cycle en médecine et 23,5 % étaient résidents. Nous avons constaté que la pandémie a négativement affecté le bien-être des apprenants à tous les niveaux de formation et que les effets précis sur les étudiants pouvaient varier en fonction de la réponse de leur programme à la situation sanitaire. CONCLUSIONS: Les étudiants en médecine et en sciences de la santé signalent une détérioration de leur bien-être liée aux programmes et aux systèmes dans lesquels ils évoluent, auxquels s'ajoute le fardeau supplémentaire de la pandémie de la COVID-19. Il conviendrait de définir les interventions futures sur la base d'un cadre holistique du bien-être des apprenants tout en adoptant une réflexion systémique pour appréhender l'entrecroisement entre individus, programmes et systèmes. Reconnaître l'importance de l'équité, de la diversité et de l'inclusion, favoriser la sécurité psychologique et assurer la participation active des apprenants dans leur parcours pendant la pandémie et au-delà, constituent des éléments clés dans l'élaboration d'interventions au profit de leur bien-être.

Wellbeing and mental health amongst medical students in Canada.

Across the world there is significant evidence that medical students have high levels of mental ill-health and psychological distress with subsequent concerning effects on personal and occupational functioning. In Canada, recent studies have demonstrated worrying levels of burnout and depressive symptoms among practising doctors. In common with other countries, Canadian medical students are also subject to a high-pressure environment - with long clinical weeks and significant stressors - and these soon-to-be doctors have been previously shown to already demonstrate high levels of burnout. We surveyed 69 medical students at the Cumming School of Medicine, Calgary regarding their wellbeing and mental health. 26% of the students had been diagnosed with a mental health condition prior to medical school, while 36% reported currently seeing a professional regarding their mental ill-health, with anxiety disorders forming the most commonly reported conditions. 83% reported their studies as a significant source of stress. 22% tested as CAGE positive and a number of students reported using other substances. 70% of medical students met specified case criteria for exhaustion on the Oldenburg Burnout Inventory. These findings speak to the need for access to mental health services, evidence-based individual counselling, and inclusive activities that fit within organisational frameworks to better improve the mental health and wellbeing of medical students in Canada.

Podcasting in medical education: can we turn this toy into an effective learning tool?

Advances in information technology have changed how we deliver medical education, sometimes for the better, sometimes not. Technologies that were designed for purposes other than education, such as podcasting, are now frequently used in medical education. In this article, the authors discuss the pros and cons of adapting existing technologies for medical education, caution against limiting evaluation of technologies to the level of rater satisfaction, and suggest a research agenda for formally evaluating the role of existing and future technologies in medical education.

Simulation training improves diagnostic performance on a real patient with similar clinical findings.

BACKGROUND: Training on a cardiopulmonary simulator improves subsequent diagnostic performance on the same simulator. But data are lacking on transfer of learning. The objective of this study was to determine whether training on a cardiorespiratory simulator improves diagnostic performance on a real patient. METHODS: We randomly allocated first-year medical students at the University of Calgary to simulator training in one of three clinical scenarios of acute-onset chest pain: pulmonary embolism with right ventricular strain but no murmur, symptomatic aortic stenosis, or myocardial ischemia causing mitral regurgitation. Simulation sessions ran for 20 min, after which participants had a standardized debriefing session and reviewed the physical findings. Immediately following the training sessions, students assessed the auscultatory findings of a real patient with mitral regurgitation. Our outcome measures were accuracy of identifying abnormal auscultatory findings and diagnosing the underlying cardiac abnormality (mitral regurgitation). RESULTS: Eighty-six students participated in the study. Students trained on mitral regurgitation were more likely to identify and diagnose these findings on a real patient with mitral regurgitation than those who had trained on aortic stenosis or a scenario with no cardiac murmur. The accuracy (SD) of identifying clinical features of mitral regurgitation for these three groups was 74.0 (36.4) vs 56.2 (34.3) vs 36.8 (33.1), respectively (P = .0005), and for diagnosing mitral regurgitation, the accuracy was 68.0 (45.4) vs 51.6 (50.0) vs 29.9 (40.7), respectively (P = .01). CONCLUSIONS: Simulator training on mitral regurgitation increases the likelihood of diagnosing this abnormality on a real patient.